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The  U.S.  Department  of  Defense  (DoD) 
is  attempting  to  field  warfighter  capa¬ 
bilities  faster,  within  a  tighter  budget, 
and  with  less  impact  on  the  operational 
users.  Concurrently,  the  DoD  is  also 
moving  to  a  net-centric  environment  enabling  infor¬ 
mation  and  service  collaboration  across  the  global 
information  grid,  requiring  new  and  innovative  ways  to 
conduct  operational  tests  (OT).  Joint  Interoperability 
Test  Command  (JITC)  testers  are  utilizing  some  of 
these  new  collaboration  capabilities  provided  by  the 
Defense  Information  Systems  Agency  to  reduce  costs, 
increase  flexibility,  reduce  warfighter  support  require¬ 
ments,  and  improve  quality  during  OTs  for  net-centric 
systems. 

Net-centric  systems  and  services  typically  consist  of 
strategically  located  applications  and/or  authoritative 
data  sources,  which  provide  capabilities  to  users  via  the 
unclassified  and  classified  DoD  networks.  These 
systems  may  or  may  not  include  client-side  applica¬ 
tions,  but  many  require  nothing  more  than  access 
permissions  and  a  traditional  web  browser  to  use.  Users 
access  the  capabilities  from  their  duty  locations  or  from 
wherever  they  have  connectivity.  One  example  of  such 
a  system  is  the  Global  Combat  Support  Systems- 
Combatant  Command/Joint  Task  Force  (GCSS-CC/ 
JTF),  which  provides  logistics  situational  awareness, 
decision  support,  and  visualization  tools  to  combatant 
commands  and  Joint  Task  Force  commanders. 

A  net-centric  system  OT  requires  genuine  opera¬ 
tional  users  to  exercise  the  system  under  test  in  a 
manner  consistent  with  its  intended  use  to  support  a 
variety  of  user  missions.  This  is  normally  achieved  by 
users  accessing  the  system  and  executing  a  series  of 
operationally  realistic  scenarios  while  data  collectors 
observe  and  record  the  activities  of  the  users.  This 
requires  extensive  data  collector  travel  to  operational 
user  locations  and  a  very  structured  and  rigid  schedule 
for  warfighter  participation.  The  flexibility  to  accom¬ 
modate  unanticipated  users  at  new  locations  or  to  make 
last-minute  schedule  changes  is  significantly  hindered 
by  having  to  physically  colocate  data  collectors  at  each 
user  location,  mainly  because  of  travel  arrangements. 


visit  requests,  and  resource  allocations  and  reservations. 
Time  and  resources  are  also  wasted  when  users  are  not 
available  as  scheduled,  and  data  collection  opportuni¬ 
ties  are  lost  when  unanticipated  users  became  available 
at  the  last  minute. 

Net-Centric  Enterprise  Services  offers  collaboration 
technologies  through  Button  One  (E-CoUab  Center) 
and  Button  Two,  Defense  Connect  Online  (DCO) 
tools.  Either  tool  provides  the  ability  to  conduct 
collaboration  events  such  as  meetings  or  document 
reviews  and  offers  voice  over-internet  protocol  (VOIP) 
services,  video  (webcam),  and  the  ability  to  share 
documents  and  a  computer  screen  over  the  global 
information  grid.  Both  tools  enable  varying  degrees  of 
control  by  the  meeting  organizers  and  are  customizable 
to  maximize  collaborative  efforts.  The  ability  to  record 
these  meetings  and  the  VOIP  and  screen-sharing 
capabilities  is  the  keystone  attribute  that  enabled  JITC 
to  utilize  these  tools  for  OT. 

JITC  incorporated  DCO  as  part  of  their  test 
strategy  in  conducting  the  GCSS-CC/JTF  v7.0  OT. 
JITC  orchestrated  activities,  collected  data,  and 
observed  operational  users  remotely  using  DCO- 
provided  capabilities  over  the  classified  network.  The 
testers  were  located  at  the  JITC  Headquarters  in 
Arizona  while  the  various  users  participated  from  their 
desktops  at  the  following  locations: 

•  Defense  Information  Systems  Agency  Sky  7, 
Virginia; 

•  Joint  Forces  Command,  Virginia; 

•  Central  Command,  Florida; 

•  Pacific  Command,  Hawaii; 

•  Special  Operations  Command,  Florida; 

•  Transportation  Command,  Illinois;  and 

•  The  Pentagon,  Virginia. 

The  major  shift  in  test  conduct  enabled  users  to 
participate  from  their  actual  classified  workstation, 
which  minimized  the  impact  on  the  users  and 
increased  the  scope  of  operational  environment  avail¬ 
able  for  testing. 

Another  benefit  allowed  stakeholders  such  as  the 
Director,  Operational  Test  6c  Evaluation,  and  the 


30(2)  •  June  2009  193 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

JUN2009  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2009  to  00-00-2009 

4.  TITLE  AND  SUBTITLE 

Operational  Testers  Use  Net-Centric  Tools  to  Test  Net-Centric  Systems 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Joint  Interoperability  Test  Command, P.O.  Box  12798, Fort 

Huachuca,AZ, 85670-2798 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  aS 

unclassified  unclassified  unclassified  Report  (SAR) 

4 

Standard  Form  298  (Rev.  8-98} 

Prescribed  by  ANSI  Std  Z39-18 


Smith  &c  Winterle 


Figure  1.  Defense  Connect  Online  Test  Administrator  Screen.  The  test  administrator  opens  individual  meetings  for  each  participant/ 
user.  The  user  shares  his/her  screen  while  using  the  Global  Combat  Support  Systems-Combat  Command/Joint  Task  Force.  Using  the 
record  function,  we  eliminate  the  need  for  screen  capture  instrumentation. 


Program  Management  Office  representatives  to  observe 
the  testing  via  DCO  from  their  classified  desktop  at  their 
respective  locations.  In  one  case,  a  conference  room  was 
set  up  to  show  the  DCO  session  on  a  wall  screen  to 
facilitate  group  discussion  and  participation. 

Testers  organized  and  created  the  DCO  sessions  for 
each  user  to  enable  easy  and  consistent  identification 
using  the  DCO  web  interface.  Users  were  then  able  to 
share  their  screens  when  given  the  “presenter”  role  by 
the  meeting  hosts.  Observers  were  limited  to  moni¬ 
toring  and  chat  functions  to  help  prevent  them  from 
unintentionally  interrupting  the  test  process.  Testers 
scheduled  sessions  with  users  as  they  were  available, 
then  distributed  the  session  schedules  and  the  DCO 
universal  resource  locator  to  all  stakeholders  for  each 
event.  The  same  user  sessions  were  used  for  the 
duration  of  the  test  to  consolidate  all  user  recordings 
under  the  same  universal  resource  locator  for  later 
reference.  The  testers  posted  downloadable  operational 
scenarios,  surveys,  test  incident  report  forms,  and  other 
test  documents  through  the  DCO  file  share  pod.  This 
made  all  test  documentation  instantly  available  to  all 
users  when  needed  and  ensured  a  common  tool  set 
among  users  and  testers. 

Users  shared  their  desktops  while  using  GCSS-CC/ 
JTF  to  perform  test  scenarios,  which  enabled  everyone 
monitoring  the  session  to  observe  as  the  users 
performed  various  activities  with  the  GCSS-CC/JTF 


system.  Figure  1  is  an  example  of  the  test  administra¬ 
tor’s  screen.  Using  the  session  record  function  of 
DCO,  testers  eliminated  the  need  for  external  screen 
capture  tools  or  instrumentation. 

Figure  2  is  an  example  of  the  data  collectors’  screens 
while  monitoring  and  capturing  the  actions  of  multiple 
users.  The  data  collector  can  monitor  one  or  more  users 
at  a  time.  The  data  collector  starts  a  separate  session/ 
test  event  as  users  become  available.  While  monitoring, 
the  data  collector  can  follow  along  as  the  user  steps 
through  the  scenarios. 

The  Test  Administrator  can 

•  monitor  or  report  scenario  completion  by  a  user, 

•  field  questions  by  users, 

•  run  a  test  log, 

•  gather  timeliness  data  from  users  in  real  time,  and 

•  monitor  test  incident  problem  reports  (TIPRs) 
and  log  calls  to  help  desk. 

In  a  separate  DCO  session,  the  Test  Administrator 
can 

•  conduct  daily  hotwashes  and 

•  run  a  data  authentication  group. 

Testers  controlled  the  test  events  using  the  'FOIP 
and  text  chat  functions.  Testers  used  these  functions  to 
provide  the  user  with  instructions  and  to  answer 
questions.  The  tester  used  the  recording  feature  to 
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Figure  2.  Test  administrator  screen  monitoring  multipie  users. 


capture  video  and  VOIP  data  for  each  test  event, 
allowing  later  replay  of  results  and  the  capturing  of 
error  messages,  user  errors,  or  any  significant  event  for 
further  analysis  or  troubleshooting  hy  the  test  team 
during  data  reduction  or  by  the  developer  in  identify¬ 
ing  potential  problems. 

There  were  obvious  benefits  using  the  DCO 
collaboration  tool  in  the  execution  of  the  GCSS-CC/ 
JTF  test,  with  cost  reduction  benefits  being  the  most 
measurable.  By  executing  the  test  remotely,  JITC  saved 
approximately  $60,000  in  labor  and  travel  costs  of  the 
$140,000  estimated  for  test  execution.  Employing 
centralized  data  collection  increased  test  flexibility  for 
accommodating  unanticipated  users/locations  and  last- 
minute  schedule  changes  by  eliminating  the  need  to 
physically  colocate  data  collectors  with  each  user.  The 
ability  to  reschedule  enabled  unanticipated  users  to  be 
easily  accommodated  into  the  test  event  or  enabled 
planned  users  to  reschedule  as  needed  because  of  test 
schedule  changes  or  user  availability.  The  ease  and 
speed  of  rescheduling  and  adaptability  to  last-minute 
changes  resulted  in  the  conservation  of  resources  and 
overall  timesaving,  with  no  loss  of  data  collection. 

Test  cost,  time,  and  resources  were  reduced  by 
employing  multiple,  simultaneous  DCO  sessions, 
which  enabled  one  data  collector  to  monitor  multiple 
users/test  events  at  the  same  time  from  his/her  desktop 
at  a  centralized  location.  Users  no  longer  had  to  spend 
time  taking  and  saving  screenshots  to  support  data 
collection  needs.  The  DCO  session  recordings  cap¬ 
tured  the  user  actions  and  the  system  responses 
throughout  the  test,  resulting  in  a  visual  record  of  all 
activities  and  all  anomalies.  The  ability  to  review  the 
recordings  assisted  with  authenticating  test  data  by 


verifying  user  inputs  at  the  time  of  recorded  system 
errors.  The  Program  Management  Office,  having 
access  to  the  recording,  could  assist  in  determining 
the  possible  root  causes  of  any  system  errors  found  and 
in  developing  the  necessary  fixes. 

To  ensure  success  when  using  collaboration  tools  for 
testing,  the  following  preparations  should  be  in  place. 

•  Ensure  maximum  network  connectivity  during 
the  test  window: 

o  ensure  system  administrator  support  is  avail¬ 
able  outside  of  normal  duty  hours,  and 
o  rehearse  actions  on  disconnection  from  the 
network. 

•  Ensure  users  have  headsets  and  can  use  them  on 
their  classified  network: 

o  allow  hands-free  communication  with  the 
data  collector,  preventing  the  need  to  switch 
applications  to  use  a  chat  tool. 

•  Minimize  the  number  of  simultaneous  users  with 
each  data  collector: 

o  data  collectors  can  better  focus  on  user 
questions  and  actions,  and 
o  multiple  or  larger  monitors  can  assist  in 
maximizing  data  collection  if  multiple  users 
must  be  monitored  by  a  single  data  collector. 

Utilizing  the  Net-Centric  Enterprise  Services 
collaboration  tools  in  executing  the  GCSS-CC/JTE 
OT  reduced  cost  by  approximately  $60,000,  increased 
the  ability  to  incorporate  schedule  or  other  changes, 
and  maximized  data  collection  opportunities.  The 
biggest  advantage  was  the  improvement  of  test  data, 
in  quality  and  quantity,  by  using  the  recording 
function  and  facilitating  the  warfighter’s  ability  to 
participate. 

Operational  test  community  strategies  must  explore 
new  and  existing  technologies,  such  as  the  Net-Centric 
Enterprise  Services  collaboration  tools,  to  keep  pace 
with  accelerated  acquisition  programs  that  are  trying  to 
field  improved  capabilities  to  the  warfighter  commu¬ 
nity.  □ 

Dewain  C.  Smith  is  a  joint  testing  ojjicer  at  the 
Operational  Testing  Division,  Joint  Interoperability  Test 
Command  (JITC),  Fort  Huachuca,  Arizona.  Mr.  Smith 
has  more  than  13  years  testing  experience  and  is  currently 
the  operational  test  action  oJJlcer  for  the  Net-Centric 
Enterprise  Services  program.  His  experience  includes 
6  years  at  JITC  directing  operational  tests  on  major 
acquisition  information  systems  and  more  than  7  years 
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operational  test  experience  while  assigned  to  the  Air  Force 
Operational  Test  and  Evaluation  Center.  He  has 
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served  a  total  of  14  years  active  duty  in  the  U.S.  Army, 
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The  evolution  of  network  centric  operations 
for  joint  military  forces  has  challenged 
application  developers,  network  architects,  and 
operators  as  our  forces  come  to  increasingly 
rely  on  access  to  the  information  and 
collaboration  that  network  centricity  brings 
to  the  battlespace.  The  increased  use  and 
complexity  of  these  systems  presents  new 
challenges  in  adequately  testing  and  evaluating 
the  tools,  applications,  and  architectures 
supporting  military  networks.  Among  the 
topics  planned  for  this  intense  workshop-like 
conference  are: 

•  Testing  in  SOA  Environment 
•  Sensor  to  Shooter  Testing  in  an 
Networked  Environment 
•  Testing  for  Network  Centric  Capabilities 
in  Support  of  Irregular  Warfare 
•  Information  Assurance  Testing  in  Network 
Centric  Environment 
•  Testing  in  Support  of  Cyber  Operations 
•  The  Use  of  Modeling  and  Simulation  in 
Net  Centric  Testing 


Abstract  Submission 

The  deadline  for  the  submission 
of  abstracts  is  July  17,  2009.  Details 
on  the  submission  process  can  be 
found  at  vmw.itea.org  along  with 
descriptions  for  each  topic.  Please 
contact  Terry  McKearney  at 
terry.mckearney@therangergroup.com 

Exhibits 

Your  company  or  government 
organization  will  want  to  take 
advantage  of  the  premium  space  that 
is  available  for  you  to  display  and 
demonstrate  products  and  services  for 
the  test  and  evaluation  community. 

To  obtain  an  application  to  exhibit 
or  to  see  the  floor  plan,  visit 
www.itea.org 

Fees 

Available  on  the  ITEA  website. 

Sponsorship 

Fours  levels  of  sponsorship  are 
available  for  your  company  to 
participate  in:  Platinum  $2500, 

Gold  $1000,  Silver  $500  and 
Bronze  $250.  For  more  information 
please  visit  www.itea.org 


Accommodations 

An  ITEA  room  block  has  been 
established  for  those  attending  this 
conference  with  a  special  corporate 
rate  of  $180  and  government  rate 
of  $147  at  the  Paradise  Point  Resort, 
San  Diego,  California  92109. 

We  encourage  you  to  make  your 
hotel  reservations  early  by  calling 
800.344.2626.  Deadline  for 
reservations  is  October  5,  2009. 

For  more  information  on  the  hotel, 
please  visit  wvm.paradisepoint.com 

Golf  Tournament 

Join  us  for  our  4th  Annual  Greater 
San  Diego  Chapter  ITEA  Golf 
Tournament  at  NAS  North  Islands 
Sea  ‘N  Air  Golf  Course.  Hole 
sponsorships  are  available  for  $250. 
Contact  Mr.  Brian  Bowden  at 
bbowden@epsilonsystems.com 
or  Mr.  Jack  Sears  at 
jsears@epsilonsystems.com  for 
more  information.  You  can  also  visit 
the  San  Diego  Chapter  website 
at  wvm.ITEA-SD.com  for  more 
information  and  to  register. 


Tour 

The  test  facilities  at  the  Navy’s  Space 
and  Naval  Warfare  Systems  Center, 
Pacific.  The  Center  is  the  Navy’s 
primary  command  and  control 
laboratory,  involved  in  developing 
both  Navy  and  joint  network  centric 
capabilities.  The  tours  will  give  a 
“hands  on”  context  to  the  workshop’s 
proceedings. 
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